1.5D Parallel Sparse Matrix-Vector Multiply

There are three common parallel sparse matrix-vector multiply algorithms: 1D row-parallel, 1D column-parallel and 2D row-column-parallel. The 1D parallel algorithms offer the advantage of having only one communication phase. On the other hand, the 2D parallel algorithm is more scalable due to a high level of flexibility on distributing fine-grain tasks, whereas they suffer from two communication phases. Here, we introduce a novel concept of heterogeneous messages where a heterogeneous message may contain both input-vector entries and partially computed output-vector entries. This concept not only leads to a decreased number of messages but also enables fusing the inputand output-communication phases into a single phase. These findings are utilized to propose a 1.5D parallel sparse matrix-vector multiply algorithm which is called local row-column-parallel. This proposed algorithm requires local fine-grain partitioning where locality refers to the constraint on each fine-grain task being assigned to the processor that contains either its input-vector entry, or its output-vector entry, or both. This constraint, nevertheless, happens to be not very restrictive so that we achieve a partitioning quality close to that of the 2D parallel algorithm. We propose two methods for local fine-grain partitioning. The first method is based on a novel directed hypergraph partitioning model that minimizes total communication volume while maintaining a load balance constraint as well as an additional locality constraint which is handled by adopting and adapting a recent and simple yet effective approach. The second method has two parts where the first part finds a distribution of the inputand output-vectors and the second part finds a nonzero/task distribution that exactly minimizes total communication volume while keeping the vector distribution intact. We conduct our experiments on a large set of test matrices to evaluate the partitioning qualities and partitioning times of these proposed 1.5D methods.